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SUMMARY 


A computer-based tarn of, “K^in" 
t Snd^SV sTnctaL’ char- 

JSSStf MSS* of the concept on the author's research and creauve 
work is illustrated through two examples. 


INTRODUCTION 

developed into an authentic and accepted new art form (ret. l). 

In 1967, on learning that saS^n-l^c t“hnoiogfwo“d 

SSai#i«SS!5S5SSL 

^S^^^psassssA 

the same as the one proposed and described here (ref. 2). 

Before oroviding a systematic oudine of synesthedc art and its requirements, it may be help- 
M if I%tarite what I mean by synesthenc ar, and wha, I visualize when ustng the 
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expression. The best way to do this vividly and expeditiously calls for a small exercise in 
imagineenng. 

Think of an empty transparent block of space about the size of a 19-in. computer-graphic 
co or monitor, its depth the same as its height Fill this space with a collection of floating objects 
that vary considerably in shape and size, some small and spherical like marbles, others larger and 
more irregular in shape. Then add something quite different to the mix, something like a luminous 
c ou or foggy mist. Endow this combination of solid forms and vaporous intangibles with col- 
ors, textures, patterns, shadows, and other attractive qualities and attributes. 

At this point, set the ensemble in motion, into a choreography of disappearing and reappear- 
ing, swelling and contracting, disintegrating and reassembling, changing one into another and 
back, and into arrays of identical objects that move choreographically to the distinctive sounds of 
computer music. And note that the sounds are fully as spatialized as the visual material, with many 
ot them movmg m precise spatial synchrony with them. 3 

Though the dominant effect is more abstract than realistic, there are hints of the real world 
here and there. Whether abstract, realistic, or something in between, the objects pass eerily 
throughone another, completely unhindered by visible mechanical or electrical assistance. Aspects 
of painting, sculpture, photography, cinema, and dance fuse into an ambience of near trans- 
parency, with objects apparently farthest from the eye nearly as visible as those that are near. With 
forms melting into air and air mto forms, the overall effect, despite the prevailing three- 
dimensionality, is as pictorial-even as "painterly"-as it is sculptural. And because of the patterned 

and formalized movement, the affinity with choreography and the dance is as obvious as the con- 
nection with painting and sculpture. 

f J heSe ^ginaiyevents in an imaginary block of space are as far from the synesthetic art of 
the future as they are from any method of three-dimensional (3-D) projection available today. Yet 
with only minimal trouble and expense, the color monitor of an Atari 1040 ST personal computer 

mt ° a 1 not - to °- crud j e approximation of the imaginary block through the purchase 
of a set of liquid crystal stereo goggles (ref. 3). The Atari is low resolution. A more expensive 
stereographic system with higher resolution, however, if adapted to a large-format video 

system, could expand the block and the events within it to a scale of 6 x 8 ft or more 

hWi7iVnI! n d y ’ lf , my cor f ldence m * e futur e of synesthetic art is justified, the scale of the 
block wtil be measured in yards rather than feet; the quality of "reach out and touch it" realism will 

^overwhelmingly convincing, and the varied happenings within the huge block of space will be 
correspondingly impressive (ref. 5). The computer-based method of 3-D projection that can 

f n C ti^titi^SSifr^ re f 1Sm ° n -Ki C V SC ? C iS What iS meant by "supermedium" mentioned 

insmmounlaWe h ° 0graphy We know today > the Stations of both are just as likely^ prove 

In order to stress that synesthetic art is as much concerned with sound as it is with pictorial 

k ?r s (a " d 4—. performance, and narrative a" 

die block of spatial activity wdl henceforth be referred to as an "event space." It could just as wed 
however, be called a stereo event space," in acknowledgment that 3-D projection by computer 
stereographies, despite its limitations, will probably incubate development of synesthetic art for 
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GENERAL FEATURES OF SYNESTHETIC ART 


Svnesthetic art has four essential features that determine the design and operation of the 
computer-based system and how it is used to create synesthetic art. These features refer to (1) the 
comprehensive multimedia character of synesthetic art, a feature Aat calls on the systern to either 
capture or simulate a wide variety of attributes and materials, both visual and acoustic, <town 
many different forms of art; (2) the bimodal spatial synesthesia of image and sound, a feature that 
enables the system to superimpose visual and acoustic elements and move them together in e 
illusory-projective event space; (3) the aesthetically integrated character of synesthetic art, a feature 
that calls on the system to assist in organizing these disparate materials into a close-knit synesthe 
unity (an option, not a requirement imposed on users of the system); and (4) the extremely 
integrated and task-oriented character of the system itself, a feature that calls on the developers of 
the system and its software to take full advantage of the computer's ability to capture, generate, 
process, and spatially manipulate both images and sounds by drawing upon resources as diverse as 
computer graphics, image processing, computer music, and artificial intelligence, among the many 

germane fields and disciplines. 

The block diagram in figure 1 represents all four of these features in a general way. More 
concretely, however, it also represents the five major blocks of software comprising the entire 
synesthetic package of programs, along with the quite specific requirements associated with each 
of these blocks The diagram conforms to the standard format for such graphic representations, 
with input at the top, output at the bottom, and everything associated with the ongoing manipula- 
tion and control of the total mass of visual and acoustic information presented in the large central 

panel, coded in light grey. 


REQUIREMENTS OF THE SYNESTHETIC SYSTEM 


Some of the more specific features of synesthetic art can be gleaned from this summary of the 
requirements imposed on the computer system, because many of the features, functions, and 
requirements of the system provide mirror reflections of synesthetic art itself. 


Realism 

During the start-up stage of synesthetic art, a high degree of realism is hardly an achievable, 
or even desirable, objective. From the very beginning, however, some use of low-resolution, 
generalized forms of realism are necessary, first for aesthetic variety and interest, and second as 
steps in the direction of the narrative and dramatic realism associated with the long-term full 
multimedia potential of synesthetic art. As an aspect of this eventual development, the degree of 
near-perfect realism should be such that an observer peering casually into the event space might 
easily fail to distinguish between the projected image of an object and the actual object itself. This 
can be thought of as the ultimate "Turing test" for 3-D projection, a level of "reach-out-and-touch- 
it" realism that may never be fully achieved, but that is useful nonetheless as an unambiguous 
standard and objective for ongoing research and invention (ref. 6). 
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Realism in synesthetic art, in whatever degrees and varieties it appears, will be achieved 
through a method of real-world image capture that is basically photographic, whether involving 
video, cinematography, or some other method. Or it will be achieved within the computer itself 
through image synthesis "from scratch," using mathematical and algorithmic techniques along the 
lines of solids modeling and ray tracing. Or the realism will be achieved through combination of 
both of the preceding, or perhaps through a method yet to be developed. A key requirement would 
seem to be a method of 3-D capture that digitizes the information as it is acquired, facilitating its 
transmission into the computer and submission to the myriad form transformation operations basic 
to this concept of synesthetic art. 


Abstraction 

The second requirement shifts away from realism to the opposite end of the stylistic spectrum 
in demanding that the system supply an endless variety of visual qualities and attributes having as 
much to do with abstract art as with realism. These attributes, which are at the core of synesthetic 
software along with objects they enhance, pertain to such basic elements as form, shape, color, 
texture, pattern, tone, translucency, hard and soft edges, optical distortions, etc. Ideally, any of 
the styles and iconography associated with 20th century visual art and its media-starting with 
painting and sculpture, but also including photography, printmaking, computer graphics, computer 
animation, video art, abstract film, laser sculpture, and light art-should be capable of being simu- 
lated and, if necessary, translated into an effective 3-D equivalent idiom for integration into the 
synesthetic mix. In time as the synesthetic software package expands, synesthetic artists should be 
able to work in virtually any style conceivable, with no constraints other than those self-imposed 
for expressive or aesthetic reasons. The objects mentioned in the Introduction, and their mutations 
as arrays, regions, and total event spaces, are represented in the block diagram under the general 
heading of "visual/spatial components." 


A Choreography of Change and Motion 

The third requirement of the synesthetic system and its software pertains to the choreographic 
aspect of synesthetic art and to the ability of the system to adapt its visual elements to interesting 
scenarios of change and movement within the event space. This time/dynamic component is 
clearly choreographic in character, whether actual dancers are projected into the event space, or 
whether the "dancers" consist of abstract shapes, colors, textures, or wisps of smokey ephemera 
moving about and through one another. 

This choreography has three aspects. The first is a choreography of change associated with 
such terms as mutation, permutation, transformation, and metamorphosis; this has a topological 
aspect as well. The second is a choreography of movement in space, a shift from here to there, or 
of continuous movements over looping and interweaving paths of motion within the event space. 
(See the panel labeled "object motion and motion paths" in the block diagram.) And the third 
imposes a choreographic aspect on the timing of the change and motion events, which can acceler- 
ate and decelerate, and involve modulated shifts of timing as complex and subtle as the graceful 
movement of a ballerina, whose art consists as much in the timing of a movement as in the sculp- 
tural shape and arc of the movement itself. (See the panel labeled "time/change/motion synesthet- 
ics" in the block diagram.) 
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Music and Sound 


Just as the visual/spatial aspect of synesthetic art is able to draw upon and enlarge the entire 
body of resources of computer graphics, the musical/acoustic aspect is able to do the same within 
the closely associated fields of electronic and computer music. Im^rtant among these resources 
are the many customized interactive devices (keyboards, pedals, sliders, dials, the insertion and 
extraction of floppy disks) developed for composing and improvisingcomputer music Also 
important is the fact that sounds, like images, can be either captured from the real world by micro- 
phones, or synthesized through electronic or digital techniques (ref. 7). 

Most important is the computer spatialization of sound that is so central to this concept of 
bimodal synesthetic art. Evidence is plentiful that the existing, well-proven technique of spattaliz- 
ing sounds by computer is growing in use and aesthetic effectiveness. For exampk. ji^t wthin 
the past year, a spatialized composition of computer music was incoiporated into a 45-ft open-form 
sculptural construction as a bimodal mix that is almost borderline synesthetic (ref. 8). In fact, 
the sculpture itself, which is completely immobile, were kinetic in some way, and if the spatialized 
sounds interacted meaningfully with the kinetic aspect of the structure, the work might approach 

the synesthetic. 


Production and Performance 

The system and its software must provide its users with the means to work in a variety of 
modes for creating many different kinds of synesthetic art In addition to working directly and 
interactively with the system, the user should be able to take advantage of intelligent robotic and 
quasi-robotic support when it is needed for a specific purpose-i.e., a fast-moving improvisation in 
which the performer(s) could not possibly keep everything in hand without inteUigent robotic sup- 
port from the system. This robotic-type support is not only helpful, it is absolutely indispensable 
when the system is sustaining an ongoing "hands-ofT performance, a special way of using c 
system that, depending on the inclination of the user, may involve either intermittent, frequent, or 
constant intervention into what the system is doing. (If the intervention is constant, the user has 
switched by definition into the fully interactive mode.) 

Within these automated productions, an important subset is the transductive mode, yet 
another hands-off situation that essentially replaces the artist as sole intervener, with interrmttent or 
ongoing interventions from a variety of sources— interventions which are continuously n^ated 
and structured by discriminating and aesthetically "sensitive" robotic components within the soft- 
ware. These intervening agencies in turn are made up of ambient energies, signals, and other 
"information" such as light, heat, sounds, vibrations, barometric pressure, brain waves, heartbeat, 
traffic pattems-many of the endless possibilities have for years been incorporated into diverse 
forms of environmental, transductive, and "systems" art, some of them computerized, most of 
them not (ref. 9). Clearly, synesthetic art produced in the transductive mode will acquire many 
specific forms for many different kinds of users and applications, all of thernso readily 
interchangeable that the distinction between an amateur and a professional performance^ will tend to 
blur (or will, that is, if the artist working with the synesthetic system wants it that way). 

Not least important is the fully robotic mode, in which the system, driven by a program Aat 
the artist has set up (or more rarely, may even have written, or possibly expanded), behaves like an 
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autonomous artist in its own right in producing either a continuous, ongoing work in the 
performance mode, or a series of individual productions in the serial-robotic mode. This fully 
robotic approach is not as far-fetched as it may seem; variations of it have been used for years by 
some of the pioneers of computer art in this country, Europe, and elsewhere (ref. 10). 


TWO PROJECTS ON THE FRINGE OF SYNESTHETIC ART 


Synesthetic art as a concept has yet to produce an actual example of the genre to discuss or 
reproduce here. Nevertheless, I have been involved with a number of projects peripheral to 
synesthetic art over the years. These can be used to illuminate the subject, but should not be mis- 
construed as examples of what is still an art of the future. From these I have selected two projects, 
the first as an example of software oriented strictly to the robotic mode, and the second for its 
combination of both interactive and robotic possibilities. 


Applications of the Serial-Robotic Mode 

An example of software capable of generating graphics in a serial-robotic mode is the largest 
and most complicated of the programs I have designed and developed to date. Called SHAPE3D, 
it was written during the middle 1970s with the help of two talented student programmers primarily 
as an experiment in the serial-robotic design of sculpture. In addition, however, the project reflects 
my long-standing conviction that the computer, in addition to its contribution to the creative aspect 
of art, also will foster a new approach to research in art theory and aesthetics. More specifically, 
the idea concerns a highly promising synergism of theory and practice between (1) the use of suc- 
cessive series of serial-robotic productions as an innovative and potentially powerful approach to 
computer-based research in art theory and the principles of design; and (2) the testing of the rules, 
principles, compositional devices, etc., generated by this research through their use in the serial- 
robotic production of various kinds of computer art. Of course, synesthetic art is obviously the 
kind of computer art with the most to gain from this valuable source of robotic intelligence con- 
cerning formal/syntactic structure-inducing algorithms and devices (refs. 11-14). 

Operating with a vocabulary of 64 modular block-like elements and a set of 30 input parame- 
ters, SHAPE3D is capable of generating serial-robotic runs of as many as 50 or 100 or more 
graphics at a time, with never a duplicate composition in any series. The six graphics comprising 
the group reproduced as figure 2 were selected from a number of different serial-robotic runs to 
demonstrate the range of variations in style that can be obtained through various settings of the 
30 parameters. The single unframed graphic in the group of six was selected from a serial-robotic 
run of 150 compositions, the best of which was chosen as a model for the complete sculpture 
shown as figure 3 (ref. 15). 


Calligraphic Stereo-Sculpture 

For 3 months during the fall of 1978, 1 collaborated with an associate on a project that used a 
StereoRealist camera to record sequences of stereoscopic light calligraphies of the kind shown in 
figure 4. Inspired by a famous Gjon Mili strobe photograph showing Picasso drawing in space 
with a pen light, we purchased the stereo camera, collected an assortment of flashlights, colored 
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gels luminous objects, and objects that could be illuminated (including a number of translucent 
plastic buckets) and set up a kind of event space in front of the stereo camera As many as eight 
successive swoops and splashes in space were superimposed on the film in the camera tocreate 
each of about 30 stereo-calligraphies. A setup not unlike the one described here would be useful 
for collecting a large repertory of paths of motion on which to graft varieties of images and sounds. 
Or a variation could be used to capture the events in toto-\he rich colors and textures along with the 
underlying paths. Or alternatively, the effects and the paths could be simulated through software, 
or through combinations of capture and mathematical synthesis (ref. 16). 

CONCLUSION 


A concept of a new form of art called synesthetic art has been described, along with the char- 
acteristics of the computer-based system required for its production. The profoundly computer- 
oriented character of this form of art informs its relevance to themes and topics such as interactive 
graphics, virtual 3-D displays and projection systems, user-system ergonomics, artificial intelli- 
gence, robotics, and telerobotics. Preparing this paper has caused me to rethink, expand, and 
clarify my thinking on synesthetic art, and has left me even more convinced of its significance and 
virtual inevitability for the future of art. The progress of computer stereographies, in particular, 
makes it especially timely to begin thinking about actual start-up projects in stereo-synesthetics-not 
just a single project, but many of them, as the task is so multifarious and the directions that can be 
taken so diverse. In addition, this paper should assure those readers involved m fields related to 
spatial displays and instrumentation that aspects of NAS A-sponsored research may have imp lca- 
tions beyond NASA itself, beyond industry, business, and other obvious areas of possible appli- 
cation. For spatial displays are relevant to art, especially to that kind of art which is computer- 
based, time-variant, synesthetic, and looks forward to what is going to happen in the next century. 
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are heard differently in different parts of the work by those circulating around and through it. 
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u Virtually all computer-based research in art is currently in stylistics, a field that traditionally 
has been focused on the exhaustive description and analysis of a particular style of painting, 
sculpture, or architecture. Under the impact of the computer, however, this information is begin- 
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ulations of the style undergoing study. Through devising shape grammars appropriate to the tar- 
geted style, this technique has been applied to the architecture of Palladio (ref. 12), to the work ot 
the Russian non-objective painter Kandinsky (ref. 13), and to that of the American abstract painter 
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iSThis modular composition was created for an exhibition of the University of Massachusetts 
at Amherst sculpture faculty held in one of the university galleries during the spring of 1978 The 
work, which measured 12 by 17 ft at the base, consisted of a subset of the complete 64-block set 
that SHAPE3D is capable of generating, manipulating, and plotting as serial-robotic compositions. 
The modular blocks, enlarged to conform to the proportions of those used in the design generated 
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16 My associate, Michael Friedman, and I alternated between creating the swaths of color-in- 
space for superimposing on the film, and handling the camera, which involved dropping a black 
cloth over the open lens while the luminous "brush" was being replaced with another for the next 
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Figure 2.- Variations of serial-robotic runs. 
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Figure 3 - Complete sculpture chosen from a serial-robotic run of 150 compositions. 


46-12 


Figure 4 - Stereoscopic light calligraphy example. 
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